Abstract
Tracr presents the ability to compile code into a Transformer program, consisting of multi-head attention and MLP blocks.
This gives us the basis to experiment with how to invert this process and map from a transformer's parameters to the program that it implements.
In my thesis we investigated how to train meta models to take the parameters of such a Transformer program as an input and return a computation graph for the program that it implements.
I pursued this project as it lays the foundations for further research into interpreting Transformer models such as ChatGPT through the use of meta models.